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(54) Traffic management and congestion control for packet-based network 



(57) In a packet-based communications network 
(100), a virtual connection is established between a 
source node (102a) and a destination node (102b), 
which may traverse one or more intermediate nodes 
(105). During the flow of packets along the virtual con- 
nection in a forward direction, from the source node 
(102a) to the destination node (102b), each node meas- 
ures the utilization of critical resources (CUF), and this 
utilization information is inserted into return packets flow- 
ing in the backward (return) direction, from the destina- 
tion node (102b) to the source node (102b). CUF is 



indicative of the maximum utilization of any resource in 
the virtual connection forward path. If the network utili- 
zation information indicates that the resources of a vir- 
tual connection are under-utilized, the submission rate 
(SIR) of packets onto the virtual connection is increased. 
However, if the network utilization information indicates 
that the resources of the virtual connection are over-uti- 
lized, the submission rate of packets onto the network at 
the source node is reduced. 
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According to the present invention, in a packet-based communications network, a virtual connection is established 
between a source node and a destination node, which may traverse one or more intermediate nodes; during the flow of 
packets along the virtual connection in a forward direction, from the source node to the destination node, each node 
measures the utilization of critical resources, and this utilization information is piggy-backed onto packets flowing in the 
5 backward (return) direction, from the destination node to the source node. 

In further accord with the present invention, the utilization information provided to the source node is indicative of 
the maximum utilization of any resource in the virtual connection forward path. 

In still further accord with the present invention, in the absence of packets flowing in the backward direction, the 
destination node generates special empty packets for transmission to the source node upon which the utilization infor- 
w mation may be piggy-backed by the intermediate nodes. 

According further to the present invention, if the network utilization information indicates that the resources of a 
virtual connection are under-utilized, the submission rate of packets onto the virtual connection is increased. However, 
if the network utilization information indicates that the resources of the virtual connection are over-utilized, the submission 
rate of packets onto the network at the source node is reduced. 
75 According still further to the present invention, the period of time between two consecutive rate adjustments of the 

packet submission rate of packets onto a virtual connection is selected to allow the first adjustment to take effect through- 
out the network before the occurrence of the second adjustment. 

According still further to the present invention, when a virtual connection becomes active, e.g. , a source node begins 
transmission of packets onto the virtual connection, other active virtual connections reduce their use of excess network 
20 resources in favor of the newly active virtual connection, thereby allowing a quick and fair allocation of resources among 
active virtual connections. Additionally, when network utilization information indicates that a reduction in the rate of 
packets submitted onto the network is necessary, virtual connections with higher use of excess network resources incur 
higher packet submission rate reductions compared with virtual connections with lesser use of excess network resources. 
In further accord with the present invention, bandwidth admission for a virtual connection is controlled in accordance 
25 with assigned service categories for the determination of whether or not packets are admitted to the network, and for 
the identification of packets that are admitted to the network with an assigned priority for determination of which packets 
will be discarded under certain congestion conditions. 

In still further accord with the present invention, credits are associated with each service category such that various 
credits are assigned to a virtual connection per measurement period, T, for admission of packets to the network during 
30 the measurement period. Each packet is assigned a credit, and during extreme congestion conditions on the network, 
packets are discarded based on the priority of the corresponding assigned credit. 

In further accord with the present invention, multiple classes of service may be designated which define the band- 
width admission, trunk queue and service discipline, and CUF measurement and SIR control for a given virtual connec- 
tion. 

35 The foregoing and other objects, features and advantages of the present invention will become more apparent in 
light of the following detailed description of exemplary embodiments thereof as illustrated in the accompanying drawings. 

Brief Description of the Drawings: 

40 Fig. 1 is a schematic block diagram of a packet-based communications network; 

Fig. 2 is a schematic block diagram of an access module contained within a source node or a destination node of 
the communications network of Fig. 1 ; 

45 Fig. 3 is a schematic block diagram of a transit module contained within a node of the communications network of 

Fig. 1 ; 

Fig. 4 is a diagram showing a packet having a header portion and a data portion; 

so Fig. 5 is a graph depicting the measurement of one network round-trip delay (RTD) of a packet in the communications 

network; and 

Fig. 6 is a graph illustrating the relationship between rate control and channel utilization factor (CUF). 
55 Best Mode for Carrying Out the Invention: 

For purposes of simplifying the description of the present invention, number abbreviations are used in the following 
description. The following table of abbreviations is provided as a reference for the reader: 



3 



1NSDOCID: <EP 0719012A2_I_> 



EP 0 719 012 A2 



ABBREVIATION 


MEANING 


ATM 


Asynchronous Transfer Mode 


CIR 


Committed Information Rate 


CMP 


Congestion Monitoring Period 


CPE 


Customer Premises Equipment 


CPU 


Central Processing Unit 


CUF 


Channel Utilization Factor 


EIR 


Excess Information Rate 


FF 


Fairness Factor 


ISR 


Initial Submission Rate 


PAD 


Packet Assembler/Disassembler 


RTD 


Round Trip Delay 


SIR 


Submission Information Rate 


VC 


Virtual Connection 
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During communications over a virtual connection, data generated by customer premises equipment 108 is provided 
via the access lines 1 1 7 to an access module 110 within a source node 102a. The data may be formatted in the appro- 
priate packet format by the customer premises equipment 108, or alternatively, the access module 110 may include a 
known packet assembler/disassembler (PAD) for properly formatting data provided by the customer premises equipment 

5 1 08 for provision to the communications network 1 00. Fig. 2 is a high level block diagram showing certain basic features 
of an access module 1 10 in a source node 102a. Referring to Fig. 2, the customer premises equipment 108 provides 
data to the access modules 1 10 on access lines 1 17. At least one dedicated virtual connection is established for each 
of the customer premises equipment 108 connected to the access module 1 10. Each of the virtual connections includes 
a queue 205, wherein packets provided by the customer premises equipment are initially stored, and a server 210, which 

w controls the provision of the packets from the queue to the transit module 1 12. As described above, if the information 
or data provided by the customer premises equipment is not in proper packet format for provision to the communications 
network, a packet assembler/disassembler 215 may be provided for formatting the data provided by the customer 
premises equipment 108 into the appropriate packet format. The various queues and servers associated with each of 
the virtual connections may be implemented by a central processing unit (CPU) which, as is known in the art, includes 

is appropriate processing and memory equipment for carrying out the functions described above and hereinafter with 
respect to the access module 110. 

Referring now to Figs. 1 and 3, within the source node 102a, the intermediate nodes 105, and the destination node 
102b, the packets associated with each virtual connection are provided to and from the interconnection networks 1 15 
by a transit module 1 1 2. Each transit module 1 1 2 is arranged for full-duplex communication between the interconnection 

20 network 115 and a network trunkline 120. In each direction within the transit module 1 12, a queue 305 is provided for 
receiving incoming packets from the interconnection network 115 or the network trunkline 120, and a server 310 is 
provided for retrieving packets from the queue 305 and providing the packets to the network trunkline 120 or intercon- 
nection network 115, respectively. 

In accordance with the present invention, when a virtual connection is established between a source node 102a 

25 and a destination node 102b, and possibly including one or more intermediate nodes 105. packets initially flow along 
the virtual connection in a forward direction, i.e., from the source node 1 02a to the destination node 102b. Within each 
node, the server 310 associated with the forward direction of the virtual connection takes a measurement of a channel 
utilization factor (CUF), which is a measure of the utilization of critical resources, e.g., bandwidth, processing power, 
etc., during transmission of packets in a forward direction of the virtual connection. This utilization information is piggy- 

30 backed onto packets flowing in the backward (return) direction of the virtual connection by the server 31 0 associated 
with the return direction of the virtual path. In the absence of return traffic, the destination node generates a special 
empty packet so that the utilization information may be provided to the source node. 

As described in greater detail hereinafter, the CUF information contained in the return packet traffic of a virtual 
connection is updated during transmission such that when the return traffic reaches the source node, the CUF information 

35 contained in the return traffic is indicative of the maximum degree of utilization indicated by any of the resources within 
the virtual connection. The CUF information provided in accordance with the present invention is provided to the source 
node in the most timely and efficient manner. The CUF information associated with the resource utilization experienced 
by a packet traveling in the forward direction does not travel with that packet all the way to the destination node. Instead, 
the current CUF information associated with a particular node is piggy-backed directly onto packets flowing in the return 

40 direction to the source node. As will be understood by those skilled in the art, in a virtual connection, the first node 
(access node) encountered by packets traveling in the forward direction is typically the critical and potentially most limiting 
node (with respect to resource utilization) encountered by the packets in the virtual connection. The CUF information 
related to the first node in the forward direction is the most timely information provided to the source node during packet 
transmission by the source node onto the virtual connection. 

45 If the CUF indicates that the resources on the virtual connection are under-utilized, the source node instructs the 

virtual connection server 2 1 0 to increase its packet submission rate into the network. However, if the CUF indicates that 
the resources on the virtual connection are over-utilized, the source node instructs the virtual connection server 210 to 
reduce its packet submission rate into the network. The period of time between two consecutive rate adjustments is 
selected to allow the first adjustment to take effect throughout the network before the occurrence of the second adjust- 
so ment. This time is selected to be equal to at least one network round-trip delay (RTD). A network round-trip delay is 
indicative of the time it takes for a packet transmitted from the source node to reach the destination node and return to 
the source node, via the virtual connection. In a lightly loaded network, a typical RTD is approximately 80 to 100 milli- 
seconds, and in a loaded network, a typical RTD is approximately 200 to 250 milliseconds. These times are used for 
example only and will vary based on the number of nodes, network connections, etc. 

55 The access modules 1 1 0 in the source node 1 02a controls the rate that packets are submitted to the network on a 
per virtual connection basis. As described hereinabove, each virtual connection has a separate queue 205 and server 
210, with the packet submission rate by the server 210 being updated as a function of network feedback. The rate at 
which packets are submitted to the network by a server 21 0 for a given virtual connection is referred to as the submission 
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on the virtual connection are over-utilized, e.g., CUF is greater than or equal to 100%, the access module decreases 
SIR. The general behavior of SIR as a function of CUF is illustrated in Figure 6. 

After a long period of inactivity, the rate matching mechanism is governed by the following relationship for an idle 
virtual connection: 

SIR = ISR 

For an active virtual connection, SIR is given by the following relationship: 



SIR i+ ^MIt^MA)(^g^(a-FF f ), C/flfj. E/fl,(S//?,xp)] 



Wherein FFj is a fairness factor, and a and p are constants selected to optimize network performance for specific traffic 
types. Typical values for a are between 0.95 and 1.0, and typical values for p are between 4.0 and 6.0. These values 
is are provided for example only, and it will be understood by those skilled in the art that these values will vary based on 
the particular implementation of the invention. 

SIR control is subject to the fairness factor (FF) to allocate resources fairly for all virtual connections, and to achieve 
quick SIR convergence during transient periods. FF is a measure of how close a virtual connections' SIR is to EIR, such 
that FF is maximum when SIR = EIR and minimum when SIR = CIR, as given below: 

20 

rSlRrCIR-i 

FF ^[-EWcm\ xy 

The parameter y is also selected to optimize network performance for specific traffic types. Typical values for y are in 
25 the rage from 0 to 0. 1 . 

FF has three main effects on SIR. First, when a virtual connection becomes active, the combination of FF and ISR 
for the newly active virtual connection forces other active virtual connections to quickly reduce their use of excess 
resources in favor of the newly-active virtual connection. This allows a quick and fair allocation of resources among 
active virtual connections. Second, when a rate reduction is necessary, virtual connections with higher use of excess 

30 resources incur higher rate reduction compared to virtual connections with lesser use of excess resources. Third, when 
the network utilization changes, FF leads to a quick SIR stabilization for all active virtual connections. 

CUF is the key factor of the traffic management system. It is the ratio of the demand for network resources to the 
total available resources. CUF is measured over a period of time, referred to as the congestion monitoring period (CMP). 
CMP is sufficiently long enough to allow an accurate statistical measurement of CUF, while at the same time, it is suffi- 

35 ciently short enough to allow several CUF measurements per RTD. The network resources being monitored are proc- 
essor power and trunk bandwidth. The CUF value reported by each node is the maximum of both processor utilization 
and trunk utilization. 

During a CMP, each transit module accumulates the total number of offered packets (both ingress and egress) for 
a given processor and the total number of offered bytes (egress only) for a given trunk. At the end of a CMP, the processor 
40 utilization is calculated as the ratio of the total number of offered packets to the maximum switching capacity of the 
processor. The trunk utilization is calculated as the ratio of the total number of offered bytes to the maximum bandwidth 
capacity of the trunk. Since CUF is a ratio of offered load to available capacity, CUF can be greater than 1 . 

Referring to Fig. 4, CUF is encoded and placed in a CUF field 415 of the packet header 402. The CUF field 415, 
for example, may represent values ranging from 1% to 1000%. In addition, a special CUF value may be generated in 
45 response to extreme network congestion situations. The special CUF value instructs the access module 110 (Fig. 1) of 
the virtual connection to immediately reduce SIR to CIR. The special CUF may be indicated by a special code placed 
in the CUF field 415. Alternatively, an additional field may be provided in the header portion 402 for the special code. 

Referring again to Fig. 1, the present invention provides a unique approach to communicating CUF values from 
network 100 to the source node 102a. Assume a virtual connection is established from source node A to destination 
so node D via intermediate nodes B and C as illustrated in Figure 1 . As packets flow from nodes A to D, all transit modules 
1 1 2 along the virtual connection measure processor utilization and trunk utilization. At the end of every CMP, each transit 
module inserts its measured CUF into all packets flowing from nodes D to A only if the measured CUF is greater than 
the CUF stored in the packet. Therefore, packets that reach node A contain the maximum CUF encountered along the 
virtual connections' forward path. The source node access module 110 uses this information to adjust the virtual con- 
55 nections* SIR accordingly. 

As described hereinabove, the present invention does not communicate network utilization information to the des- 
tination node, which in turn echoes the same information back to the source node. This approach could cause the 
network utilization information to incur a full network round-trip delay before it reaches the source node. Instead, the 
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1) bandwidth admission and enforcement at the network access point; 

2) trunk queue and service discipline; and 

5 3) CUF measurement and its application to SIR control. 

An access module 110 might implement a different criteria of bandwidth admission, bandwidth enforcement, and SIR 
control depending on the assigned class of service. For example, for a virtual connection associated with a high priority 
communications channel, a large number of committed credits may be assigned to ensure bandwidth admission. Addi- 
io tionally, a, p, and y are selected to ensure that SIR is responsive for communications of packets onto the network. 

In one embodiment of the invention, each class of service might correspond to a different queue in the transit mod- 
ules. In this case, a transit module 112 measures network resource utilization per queue, e.g., per virtual connection, 
and generates a CUF that applies to all virtual connections of the same class of service. Alternatively, one queue may 
be provided for all classes of service. It will therefore be understood that, depending on how transit queues are served, 
is the CUF measurement for a given class of service might also depend on the resource utilization of other classes of 
service that share the same trunk. 

Although the invention has been described and illustrated with respect to exemplary embodiments thereof, it should 
be understood by those skilled in the art that the foregoing, and various other additions and omissions may be made 
therein and thereto without departing from the spirit and scope of the present invention. 

20 

Claims 

1 . A system for traffic management and congestion control on a virtual connection of a packet-based communications 
network, comprising: 

25 customer premises equipment for providing packets for transmission on the virtual connection, and for receiv- 

ing packets transmitted on the virtual connection; 

a source node for accepting packets from corresponding source node customer premises equipment, and 
for transmitting the packets over the virtual connection through the packet-based communications network at a 
variable submission information rate (SIR); 
30 a destination node for accepting the packets transmitted in a forward direction over the virtual connection 

from said source node to said destination node, for providing the packets transmitted in the forward direction to 
corresponding destination node customer premises equipment connected thereto, and for transmitting return pack- 
ets in a return direction over the virtual connection from said destination node to said source node; and 

means for measuring a channel utilization factor (CUF), which is directly related to the degree of utilization 
35 of network resources, during transmission of packets in said forward direction, and for inserting said channel utili- 
zation factor in return packets travelling in said return direction; 

wherein said source node comprises means for controlling said submission information rate in response to 
said channel utilization factor contained in said return packets. 

ao 2. A system according to claim 1 , wherein said source node and said destination node comprise: 

at least one access module for interconnection with an access line connected to customer premises equip- 
ment, said access line providing a transmission path for packets between said access module and said customer 
premises equipment; 

at least one transit module for interconnection with a network trunk line; and 
45 an interconnection network for interconnecting said access modules and said transit modules. 

3. A system according to claim 2, wherein the channel utilization factor is measured at each transit module in the virtual 
connection, and wherein the channel utilization factor measured at a transit module is inserted into a return packet 
only if its magnitude is greater than the magnitude of the channel utilization factor previously inserted into the return 

so packet. 

4. A system according to claim 1 or any one of the following, further comprising: 

at least one intermediate node between said source node and said destination node in the virtual connection, 
said intermediate node including: 
55 at least two transit modules, each for interconnection with a network trunk line; and 

an interconnection network for interconnecting said transit modules. 

5. A system according to claim 2 or any one of the following, wherein said access modules comprise: 

at least one queue for queuing packets received from customer premises equipment for transmission on the 
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A system according to claim 2 or any one of the following, wherein said transit modules comprise- 

^ QUeUe for QUeUing received f rom *• interconnect network 

a trunk line queue for queuing packets received from the trunk line- neworK. 

trunkline; ^° nneCti ° n network se ™ for «^«ng packets from said interconnection network queue to the 
a trunkline server for transmitting packets from said trunkline queue to the interconnection network 

A system according to claim 1 or any one of the following, wherein 



CIR ^ ISR ^ EIR. 
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S/fl /+1 = Mfl^MAxfg^a-FF^, C/fl], E/R.(S/f? /X p)] 
netolk " 3 faimeSS feCt ° r ' ° ^ P ^ 0on -* S6,ec,ed to °P«™« P-fcm-no. of the communications 
12. A system according to claim 1 1 . wherein said fairness factor is given by the following relationship: 

wherein y is a constant selected to optimize performance of the communications network. 
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statistically committed credits corresponding to bandwidth that is statistically allocated on the communications 
network for the virtual connection; and 

excess credits corresponding to any available bandwidth of the communications network that is greater than 
the statistically allocated and minimum guaranteed bandwidth. 

5 

14. A system according to claim 13, wherein said assigned service subscription parameters are allocated to the virtual 
connection on a per measurement interval basis, wherein the number of committed credits, statistically committed 
credits and excess credits allocated to the virtual connection are updated at the end of each measurement interval, 
and wherein during a measurement interval, 

io said source node first provides a number of packets to the virtual connection corresponding to the number 

of committed credits, and marks such packets as committed packets; 

said source node next provides a number of packets to the virtual connection corresponding to the number 
of statistically committed credits, and marks such packets as statistically committed packets; 

said source node next provides a number of packets to the virtual connection corresponding to the number 
75 of excess credits, and marks such packets as excess packets; and 
said source node discards any remaining packets. 

1 5. A system according to claim 13 or 14, wherein during transmission on the virtual connection, packets are selectively 
discarded under certain network congestion situations according to a predefined protocol wherein excess packets 

20 are discarded first, statistically committed packets are discarded second, and committed packets are discarded last. 

16. A system according to claim 13 or any one of the following, wherein multiple classes of service are defined for the 
communications network, and wherein the magnitudes of said constants a, p and y. and the amount of assigned 
committed credits, statistically committed credits and excess credits are selected based on the class of service. 

25 

17. A system according to claim 2 or any one of the following, further comprising means for providing a special channel 
utilization factor in response to extreme network congestion, said source node access modules reducing the sub- 
mission information rate to the committed information rate in response to said special channel utilization factor. 

30 18. A method for traffic management and congestion control of packets traveling in a forward direction from a source 
node to a destination node on a virtual connection of a packet-based communications network, the method com- 
prising the steps of: 

transmitting the packets from the source node to the destination node on the virtual connection at a variable 
submission information rate (SIR); 
35 measuring a channel utilization factor (CUF), which is directly related to the degree of utilization of network 

resources, during transmission of packets in said forward direction; 

inserting said channel utilization factor in return packets travelling in a return direction from the destination 
node to the source node on the virtual connection; and 

adjusting said submission information rate in response to said channel utilization factor contained in said 
40 return packets. 

19. A method according to claim 18, wherein: 

submission information rate is always greater than or equal to a committed information rate (CIR); 
submission information rate is always less than or equal to an excess information rate (EIR); and 
45 submission information rate is initially set equal to an initial submission rate (ISR), wherein the relationship 

between said committed information rate, said excess information rate, and said initial submission rate is given by: 

CIR ^ ISR - EIR. 

so 20. A method according to claim 19. further comprising the steps of: 

measuring a round-trip delay as the time it take a packet to travel from the source node to the destination 
node and back over the virtual connection; and 

determining a new submission information rate (SIR i+1 ) after each round-trip delay based on the current 
submission information rate (SIRi), said connection utilization factor said committed information rate and said excess 
55 information rate, as given by the following relationship: 

S/fi, +1 = MIN[mA)^!^( a- FF f ), C/fl] , EIR,(SIR ,xp)] 
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w^isafcirness^^ 
21. A method according to claim 20. wherein said fairness factor is given by the lowing relationship: 

rSIR r CIR-, 

-herein y is a constant selected to op«mize performance of the communications network 

tionsneTo7 fo ?^^ 
network^ 

A system according to claim 22, further comprising the steps of 

intervals 9 *" ^ ip6on * *. virtual connection on a per measurement 

during a measurement interval 

said source node discards any remaining packets. ' 
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(54) Traffic management and congestion control for packet-based network 



(57) In a packet-based communications network 
(100), a virtual connection is established between a 
source node (102a) and a destination node (102b), 
which may traverse one or more intermediate nodes 
(105). During the flow of packets along the virtual con- 
nection in a forward direction, from the source node 
(102a) to the destination node (102b), each node meas- 
ures the utilization of critical resources (CUF), and this 
utilization information is inserted into return packets 
flowing in the backward (return) direction, from the des- 
tination node (102b) to the source node (102b). CUF is 



indicative of the maximum utilization of any resource in 
the virtual connection forward path. If the network utili- 
zation information indicates that the resources of a vir- 
tual connection are under-utilized, the submission rate 
(SIR) of packets onto the virtual connection is 
increased. However, if the network utilization informa- 
tion indicates that the resources of the virtual connec- 
tion are over-utilized, the submission rate of packets 
onto the network at the source node is reduced. 
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